Systems and methods to deliver point of care alerts for radiological findings

ABSTRACT

Apparatus, systems, and methods to improve imaging quality control, image processing, identification of findings, and generation of notification at or near a point of care are disclosed and described. An example imaging apparatus includes a processor to at least: evaluate first image data with respect to an image quality measure; when the first image data satisfies the image quality measure, process the first image data using a trained learning network to generate a first analysis of the first image data; identify a clinical finding in the first image data based on the first analysis; compare the first analysis to a second analysis, the second analysis generated from second image data obtained in a second image acquisition; and, when comparing identifies a change between the first analysis and the second analysis, trigger a notification at the imaging apparatus regarding the clinical finding to prompt a responsive action.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent arises from U.S. patent application Ser. No. 15/821,161,which was filed on Nov. 22, 2017. U.S. patent application Ser. No.15/821,161 is hereby incorporated herein by reference in its entirety.Priority to U.S. patent application Ser. No. 15/821,161 is herebyclaimed.

FIELD OF THE DISCLOSURE

This disclosure relates generally to improved medical systems and, moreparticularly, to improved learning systems and methods for medical imageprocessing.

BACKGROUND

A variety of economy, operational, technological, and administrativehurdles challenge healthcare facilities, such as hospitals, clinics,doctors' offices, imaging centers, teleradiology, etc., to providequality care to patients. Economic drivers, less skilled staff, fewerstaff, complicated equipment, and emerging accreditation for controllingand standardizing radiation exposure dose usage across a healthcareenterprise create difficulties for effective management and use ofimaging and information systems for examination, diagnosis, andtreatment of patients.

Healthcare provider consolidations create geographically distributedhospital networks in which physical contact with systems is too costly.At the same time, referring physicians want more direct access tosupporting data in reports along with better channels for collaboration.Physicians have more patients, less time, and are inundated with hugeamounts of data, and they are eager for assistance.

Healthcare provider (e.g., x-ray technologist, doctor, nurse, etc.)tasks including image processing and analysis, quality assurance/qualitycontrol, etc., are time consuming and resource intensive tasksimpractical, if not impossible, for humans to accomplish alone.

BRIEF SUMMARY

Certain examples provide apparatus, systems, and methods to improveimaging quality control, image processing, identification of findings inimage data, and generation of notification at or near a point of carefor a patient.

Certain examples provide an imaging apparatus including a memoryincluding first image data obtained in a first image acquisition andinstructions and a processor. The example processor is to execute theinstructions to at least: evaluate the first image data with respect toan image quality measure; when the first image data satisfies the imagequality measure, process the first image data using a trained learningnetwork to generate a first analysis of the first image data; identify aclinical finding in the first image data based on the first analysis;compare the first analysis to a second analysis, the second analysisgenerated from second image data obtained in a second image acquisition;and, when comparing identifies a change between the first analysis andthe second analysis, trigger a notification at the imaging apparatus tonotify a healthcare practitioner regarding the clinical finding andprompt a responsive action with respect to a patient associated with thefirst image data.

Certain examples provide a computer-readable storage medium in animaging apparatus including instructions which, when executed, cause atleast one processor in the imaging apparatus to at least: evaluate thefirst image data with respect to an image quality measure; when thefirst image data satisfies the image quality measure, process the firstimage data using a trained learning network to generate a first analysisof the first image data; identify a clinical finding in the first imagedata based on the first analysis; compare the first analysis to a secondanalysis, the second analysis generated from second image data obtainedin a second image acquisition; and, when comparing identifies a changebetween the first analysis and the second analysis, trigger anotification at the imaging apparatus to notify a healthcarepractitioner regarding the clinical finding and prompt a responsiveaction with respect to a patient associated with the first image data.

Certain examples provide a computer-implemented method includingevaluating, by executed an instruction with at least one processor, thefirst image data with respect to an image quality measure. The examplemethod includes, when the first image data satisfies the image qualitymeasure, processing, by executing an instruction with the at least oneprocessor, the first image data using a trained learning network togenerate a first analysis of the first image data. The example methodincludes identifying, by executing an instruction with at least oneprocessor, a clinical finding in the first image data based on the firstanalysis. The example method includes comparing, by executing aninstructing with the at least one processor, the first analysis to asecond analysis, the second analysis generated from second image dataobtained in a second image acquisition. The example method includes,when comparing identifies a change between the first analysis and thesecond analysis, triggering, by executing an instruction using the atleast one processor, a notification at the imaging apparatus to notify ahealthcare practitioner regarding the clinical finding and prompt aresponsive action with respect to a patient associated with the firstimage data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B illustrate an example imaging system to which the methods,apparatus, and articles of manufacture disclosed herein can be applied.

FIG. 2 illustrates an example mobile imaging system.

FIG. 3 is a representation of an example learning neural network.

FIG. 4 illustrates a particular implementation of the example neuralnetwork as a convolutional neural network.

FIG. 5 is a representation of an example implementation of an imageanalysis convolutional neural network.

FIG. 6A illustrates an example configuration to apply a learning networkto process and/or otherwise evaluate an image.

FIG. 6B illustrates a combination of a plurality of learning networks.

FIG. 7 illustrates example training and deployment phases of a learningnetwork.

FIG. 8 illustrates an example product leveraging a trained networkpackage to provide a deep learning product offering.

FIGS. 9A-9C illustrate various deep learning device configurations.

FIG. 10 illustrates an example image processing system or apparatus.

FIGS. 11-12 illustrate flow diagrams for example methods of automatedprocessing and image analysis to present findings at the point of carein accordance with the systems and/or apparatus of FIGS. 1-10.

FIGS. 13-24 illustrate example displays to provide output and facilitateinteraction in accordance with the apparatus, systems, and methodsdescribed above in connection with FIGS. 1-12.

FIG. 25 illustrates an example system configuration in which an imagingsystem interfaces with a broker device to communicate with a pluralityof information systems.

FIG. 26 illustrates an example system configuration in which anartificial intelligent model executes on an edge device to provide pointof care alerts on an imaging machine.

FIG. 27 illustrates an example system to incorporate and compareartificial intelligence model processing results between current andprior exams.

FIG. 28 illustrates a flow diagrams for an example method to prioritize,in a worklist, an exam related to a critical finding for review.

FIG. 29 illustrates a flow diagram for an example method to comparecurrent and prior Al analyses of image data to generate a notificationfor a point of care alert.

FIG. 30 is a block diagram of a processor platform structured to executethe example machine readable instructions to implement componentsdisclosed and described herein.

The foregoing summary, as well as the following detailed description ofcertain embodiments of the present invention, will be better understoodwhen read in conjunction with the appended drawings. For the purpose ofillustrating the invention, certain embodiments are shown in thedrawings. It should be understood, however, that the present inventionis not limited to the arrangements and instrumentality shown in theattached drawings. The figures are not scale. Wherever possible, thesame reference numbers will be used throughout the drawings andaccompanying written description to refer to the same or like parts.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings that form a part hereof, and in which is shown byway of illustration specific examples that may be practiced. Theseexamples are described in sufficient detail to enable one skilled in theart to practice the subject matter, and it is to be understood thatother examples may be utilized and that logical, mechanical, electricaland other changes may be made without departing from the scope of thesubject matter of this disclosure. The following detailed descriptionis, therefore, provided to describe an exemplary implementation and notto be taken as limiting on the scope of the subject matter described inthis disclosure. Certain features from different aspects of thefollowing description may be combined to form yet new aspects of thesubject matter discussed below.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” “the,” and “said” are intended tomean that there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements.

While certain examples are described below in the context of medical orhealthcare systems, other examples can be implemented outside themedical environment. For example, certain examples can be applied tonon-medical imaging such as non-destructive testing, explosivedetection, etc.

I. OVERVIEW

Imaging devices (e.g., gamma camera, positron emission tomography (PET)scanner, computed tomography (CT) scanner, X-Ray machine, fluoroscopymachine, magnetic resonance (MR) imaging machine, ultrasound scanner,etc.) generate medical images (e.g., native Digital Imaging andCommunications in Medicine (DICOM) images) representative of the partsof the body (e.g., organs, tissues, etc.) to diagnose and/or treatdiseases. Medical images may include volumetric data including voxelsassociated with the part of the body captured in the medical image.Medical image visualization software allows a clinician to segment,annotate, measure, and/or report functional or anatomicalcharacteristics on various locations of a medical image. In someexamples, a clinician may utilize the medical image visualizationsoftware to identify regions of interest with the medical image.

Acquisition, processing, quality control, analysis, and storage ofmedical image data play an important role in diagnosis and treatment ofpatients in a healthcare environment. A medical imaging workflow anddevices involved in the workflow can be configured, monitored, andupdated throughout operation of the medical imaging workflow anddevices. Machine and/or deep learning can be used to help configure,monitor, and update the medical imaging workflow and devices.

Certain examples provide and/or facilitate improved imaging deviceswhich improve diagnostic accuracy and/or coverage. Certain examplesfacilitate improved image reconstruction and further processing toprovide improved diagnostic accuracy.

Machine learning techniques, whether deep learning networks or otherexperiential/observational learning system, can be used to locate anobject in an image, understand speech and convert speech into text, andimprove the relevance of search engine results, for example. Deeplearning is a subset of machine learning that uses a set of algorithmsto model high-level abstractions in data using a deep graph withmultiple processing layers including linear and non-lineartransformations. While many machine learning systems are seeded withinitial features and/or network weights to be modified through learningand updating of the machine learning network, a deep learning networktrains itself to identify “good” features for analysis. Using amultilayered architecture, machines employing deep learning techniquescan process raw data better than machines using conventional machinelearning techniques. Examining data for groups of highly correlatedvalues or distinctive themes is facilitated using different layers ofevaluation or abstraction.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The term “deep learning” is a machine learningtechnique that utilizes multiple data processing layers to recognizevarious structures in data sets and classify the data sets with highaccuracy. A deep learning network can be a training network (e.g., atraining network model or device) that learns patterns based on aplurality of inputs and outputs. A deep learning network can be adeployed network (e.g., a deployed network model or device) that isgenerated from the training network and provides an output in responseto an input.

The term “supervised learning” is a deep learning training method inwhich the machine is provided already classified data from humansources. The term “unsupervised learning” is a deep learning trainingmethod in which the machine is not given already classified data butmakes the machine useful for abnormality detection. The term“semi-supervised learning” is a deep learning training method in whichthe machine is provided a small amount of classified data from humansources compared to a larger amount of unclassified data available tothe machine.

The term “representation learning” is a field of methods fortransforming raw data into a representation or feature that can beexploited in machine learning tasks. In supervised learning, featuresare learned via labeled input.

The term “convolutional neural networks” or “CNNs” are biologicallyinspired networks of interconnected data used in deep learning fordetection, segmentation, and recognition of pertinent objects andregions in datasets. CNNs evaluate raw data in the form of multiplearrays, breaking the data in a series of stages, examining the data forlearned features.

The term “transfer learning” is a process of a machine storing theinformation used in properly or improperly solving one problem to solveanother problem of the same or similar nature as the first. Transferlearning may also be known as “inductive learning”. Transfer learningcan make use of data from previous tasks, for example.

The term “active learning” is a process of machine learning in which themachine selects a set of examples for which to receive training data,rather than passively receiving examples chosen by an external entity.For example, as a machine learns, the machine can be allowed to selectexamples that the machine determines will be most helpful for learning,rather than relying only an external human expert or external system toidentify and provide examples.

The term “computer aided detection” or “computer aided diagnosis” referto computers that analyze medical images for the purpose of suggesting apossible diagnosis.

Certain examples use neural networks and/or other machine learning toimplement a new workflow for image and associated patient analysisincluding generating alerts based on radiological findings may begenerated and delivered at the point of care of a radiology exam.Certain examples use Artificial Intelligence (AI) algorithms toimmediately (e.g., with a data processing, transmission, and/orstorage/retrieval latency) process a radiological exam (e.g., an imageor set of images), and provide an alert based on the automated examanalysis at the point of care. The alert and/or other notification canbe seen on a visual display, represented by a sensor (e.g., light color,etc.), be an audible noise/tone, and/or be sent as a message (e.g.,short messaging service (SMS), Health Level 7 (HL7), DICOM header tag,phone call, etc.). The alerts may be intended for the technologistacquiring the exam, clinical team providers (e.g., nurse, doctor, etc.),radiologist, administration, operations, and/or even the patient. Thealerts may be to indicate a specific or multiple quality control and/orradiological finding(s) or lack thereof in the exam image data, forexample.

In certain examples, the AI algorithm can be (1) embedded within theradiology system, (2) running on a mobile device (e.g., a tablet, smartphone, laptop, other handheld or mobile computing device, etc.), and/or(3) running in a cloud (e.g., on premise or off premise) and deliversthe alert via a web browser (e.g., which may appear on the radiologysystem, mobile device, computer, etc.). Such configurations can bevendor neutral and compatible with legacy imaging systems. For example,if the AI processor is running on a mobile device and/or in the “cloud”,the configuration can receive the images (A) from the x-ray and/or otherimaging system directly (e.g., set up as secondary push destination suchas a Digital Imaging and Communications in Medicine (DICOM) node, etc.),(B) by tapping into a Picture Archiving and Communication System (PACS)destination for redundant image access, (C) by retrieving image data viaa sniffer methodology (e.g., to pull a DICOM image off the system onceit is generated), etc.

Deep Learning and Other Machine Learning

Deep learning is a class of machine learning techniques employingrepresentation learning methods that allows a machine to be given rawdata and determine the representations needed for data classification.Deep learning ascertains structure in data sets using backpropagationalgorithms which are used to alter internal parameters (e.g., nodeweights) of the deep learning machine. Deep learning machines canutilize a variety of multilayer architectures and algorithms. Whilemachine learning, for example, involves an identification of features tobe used in training the network, deep learning processes raw data toidentify features of interest without the external identification.

Deep learning in a neural network environment includes numerousinterconnected nodes referred to as neurons. Input neurons, activatedfrom an outside source, activate other neurons based on connections tothose other neurons which are governed by the machine parameters. Aneural network behaves in a certain manner based on its own parameters.Learning refines the machine parameters, and, by extension, theconnections between neurons in the network, such that the neural networkbehaves in a desired manner.

Deep learning that utilizes a convolutional neural network segments datausing convolutional filters to locate and identify learned, observablefeatures in the data. Each filter or layer of the CNN architecturetransforms the input data to increase the selectivity and invariance ofthe data. This abstraction of the data allows the machine to focus onthe features in the data it is attempting to classify and ignoreirrelevant background information.

Deep learning operates on the understanding that many datasets includehigh level features which include low level features. While examining animage, for example, rather than looking for an object, it is moreefficient to look for edges which form motifs which form parts, whichform the object being sought. These hierarchies of features can be foundin many different forms of data such as speech and text, etc.

Learned observable features include objects and quantifiableregularities learned by the machine during supervised learning. Amachine provided with a large set of well classified data is betterequipped to distinguish and extract the features pertinent to successfulclassification of new data.

A deep learning machine that utilizes transfer learning may properlyconnect data features to certain classifications affirmed by a humanexpert. Conversely, the same machine can, when informed of an incorrectclassification by a human expert, update the parameters forclassification. Settings and/or other configuration information, forexample, can be guided by learned use of settings and/or otherconfiguration information, and, as a system is used more (e.g.,repeatedly and/or by multiple users), a number of variations and/orother possibilities for settings and/or other configuration informationcan be reduced for a given situation.

An example deep learning neural network can be trained on a set ofexpert classified data, classified and further annotated for objectlocalization, for example. This set of data builds the first parametersfor the neural network, and this would be the stage of supervisedlearning. During the stage of supervised learning, the neural networkcan be tested whether the desired behavior has been achieved.

Once a desired neural network behavior has been achieved (e.g., amachine has been trained to operate according to a specified threshold,etc.), the machine can be deployed for use (e.g., testing the machinewith “real” data, etc.). During operation, neural networkclassifications can be confirmed or denied (e.g., by an expert user,expert system, reference database, etc.) to continue to improve neuralnetwork behavior. The example neural network is then in a state oftransfer learning, as parameters for classification that determineneural network behavior are updated based on ongoing interactions. Incertain examples, the neural network can provide direct feedback toanother process. In certain examples, the neural network outputs datathat is buffered (e.g., via the cloud, etc.) and validated before it isprovided to another process.

Deep learning machines using convolutional neural networks (CNNs) can beused for image analysis. Stages of CNN analysis can be used for facialrecognition in natural images, computer-aided diagnosis (CAD), etc.

High quality medical image data can be acquired using one or moreimaging modalities, such as x-ray, computed tomography (CT), molecularimaging and computed tomography (MICT), magnetic resonance imaging(MRI), etc. Medical image quality is often not affected by the machinesproducing the image but the patient. A patient moving during an MRI cancreate a blurry or distorted image that can prevent accurate diagnosis,for example.

Interpretation of medical images, regardless of quality, is only arecent development. Medical images are largely interpreted byphysicians, but these interpretations can be subjective, affected by thecondition of the physician's experience in the field and/or fatigue.Image analysis via machine learning can support a healthcarepractitioner's workflow.

Deep learning machines can provide computer aided detection support toimprove their image analysis with respect to image quality andclassification, for example. However, issues facing deep learningmachines applied to the medical field often lead to numerous falseclassifications. Deep learning machines must overcome small trainingdatasets and require repetitive adjustments, for example.

Deep learning machines, with minimal training, can be used to determinethe quality of a medical image, for example. Semi-supervised andunsupervised deep learning machines can be used to quantitativelymeasure qualitative aspects of images. For example, deep learningmachines can be utilized after an image has been acquired to determineif the quality of the image is sufficient for diagnosis. Supervised deeplearning machines can also be used for computer aided diagnosis.Supervised learning can help reduce susceptibility to falseclassification, for example.

Deep learning machines can utilize transfer learning when interactingwith physicians to counteract the small dataset available in thesupervised training. These deep learning machines can improve theircomputer aided diagnosis over time through training and transferlearning.

II. DESCRIPTION OF EXAMPLES Example Imaging Systems

The methods, apparatus, and articles of manufacture described herein canbe applied to a variety of healthcare and non-healthcare systems. In oneparticular example, the methods, apparatus, and articles of manufacturedescribed herein can be applied to the components, configuration, andoperation of a computed tomography (CT) imaging system. FIGS. 1A-1Billustrate an example implementation of a CT imaging scanner to whichthe methods, apparatus, and articles of manufacture disclosed herein canbe applied. FIGS. 1A and 1B show a CT imaging system 10 including agantry 12. Gantry 12 has a rotary member 13 with an x-ray source 14 thatprojects a beam of x-rays 16 toward a detector assembly 18 on theopposite side of the rotary member 13. A main bearing may be utilized toattach the rotary member 13 to the stationary structure of the gantry12. X-ray source 14 includes either a stationary target or a rotatingtarget. Detector assembly 18 is formed by a plurality of detectors 20and data acquisition systems (DAS) 22, and can include a collimator. Theplurality of detectors 20 sense the projected x-rays that pass through asubject 24, and DAS 22 converts the data to digital signals forsubsequent processing. Each detector 20 produces an analog or digitalelectrical signal that represents the intensity of an impinging x-raybeam and hence the attenuated beam as it passes through subject 24.During a scan to acquire x-ray projection data, rotary member 13 and thecomponents mounted thereon can rotate about a center of rotation.

Rotation of rotary member 13 and the operation of x-ray source 14 aregoverned by a control mechanism 26 of CT system 10. Control mechanism 26can include an x-ray controller 28 and generator 30 that provides powerand timing signals to x-ray source 14 and a gantry motor controller 32that controls the rotational speed and position of rotary member 13. Animage reconstructor 34 receives sampled and digitized x-ray data fromDAS 22 and performs high speed image reconstruction. The reconstructedimage is output to a computer 36 which stores the image in a computerstorage device 38.

Computer 36 also receives commands and scanning parameters from anoperator via operator console 40 that has some form of operatorinterface, such as a keyboard, mouse, touch sensitive controller, voiceactivated controller, or any other suitable input apparatus. Display 42allows the operator to observe the reconstructed image and other datafrom computer 36. The operator supplied commands and parameters are usedby computer 36 to provide control signals and information to DAS 22,x-ray controller 28, and gantry motor controller 32. In addition,computer 36 operates a table motor controller 44 which controls amotorized table 46 to position subject 24 and gantry 12. Particularly,table 46 moves a subject 24 through a gantry opening 48, or bore, inwhole or in part. A coordinate system 50 defines a patient or Z-axis 52along which subject 24 is moved in and out of opening 48, a gantrycircumferential or X-axis 54 along which detector assembly 18 passes,and a Y-axis 56 that passes along a direction from a focal spot of x-raytube 14 to detector assembly 18.

Thus, certain examples can apply machine learning techniques toconfiguration and/or operation of the CT scanner 10 and its gantry 12,rotary member 13, x-ray source 14, detector assembly 18, controlmechanism 26, image reconstructor 34, computer 36, operator console 40,display 42, table controller 44, table 46, and/or gantry opening 48,etc. Component configuration, operation, etc., can be monitored based oninput, desired output, actual output, etc., to learn and suggestchange(s) to configuration, operation, and/or image capture and/orprocessing of the scanner 10 and/or its components, for example.

FIG. 2 illustrates a portable variant of an x-ray imaging system 200.The example digital mobile x-ray system 200 can be positioned withrespect to a patient bed without requiring the patient to move andreposition themselves on the patient table 46 of a stationary imagingsystem 10. Wireless technology enables wireless communication (e.g.,with adaptive, automatic channel switching, etc.) for image and/or otherdata transfer to and from the mobile imaging system 200. Digital imagescan be obtained and analyzed at the imaging system 200 and/ortransferred to another system (e.g., a PACS, etc.) for further analysis,annotation, storage, etc.

The mobile imaging system 200 includes a source 202 and a wirelessdetector 204 that can be positioned underneath and/or otherwise withrespect to a patient anatomy to be imaged. The example mobile system 200also includes a display 206 to display results of image acquisition fromthe wireless detector 204. The example mobile system 200 includes aprocessor 210 to configure and control image acquisition, imageprocessing, image data transmission, etc.

In some examples, the imaging system 10, 200 can include a computerand/or other processor 36, 210 to process obtained image data at theimaging system 10, 200. For example, the computer and/or other processor36, 210 can implement an artificial neural network and/or other machinelearning construct to process acquired image data and output ananalysis, alert, and/or other result.

Example Learning Network Systems

FIG. 3 is a representation of an example learning neural network 300.The example neural network 300 includes layers 320, 340, 360, and 380.The layers 320 and 340 are connected with neural connections 330. Thelayers 340 and 360 are connected with neural connections 350. The layers360 and 380 are connected with neural connections 370. Data flowsforward via inputs 312, 314, 316 from the input layer 320 to the outputlayer 380 and to an output 390.

The layer 320 is an input layer that, in the example of FIG. 3, includesa plurality of nodes 322, 324, 326. The layers 340 and 360 are hiddenlayers and include, the example of FIG. 3, nodes 342, 344, 346, 348,362, 364, 366, 368. The neural network 300 may include more or lesshidden layers 340 and 360 than shown. The layer 380 is an output layerand includes, in the example of FIG. 3, a node 382 with an output 390.Each input 312-316 corresponds to a node 322-326 of the input layer 320,and each node 322-326 of the input layer 320 has a connection 330 toeach node 342-348 of the hidden layer 340. Each node 342-348 of thehidden layer 340 has a connection 350 to each node 362-368 of the hiddenlayer 360. Each node 362-368 of the hidden layer 360 has a connection370 to the output layer 380. The output layer 380 has an output 390 toprovide an output from the example neural network 300.

Of connections 330, 350, and 370 certain example connections 332, 352,372 may be given added weight while other example connections 334, 354,374 may be given less weight in the neural network 300. Input nodes322-326 are activated through receipt of input data via inputs 312-316,for example. Nodes 342-348 and 362-368 of hidden layers 340 and 360 areactivated through the forward flow of data through the network 300 viathe connections 330 and 350, respectively. Node 382 of the output layer380 is activated after data processed in hidden layers 340 and 360 issent via connections 370. When the output node 382 of the output layer380 is activated, the node 382 outputs an appropriate value based onprocessing accomplished in hidden layers 340 and 360 of the neuralnetwork 300.

FIG. 4 illustrates a particular implementation of the example neuralnetwork 300 as a convolutional neural network 400. As shown in theexample of FIG. 4, an input 310 is provided to the first layer 320 whichprocesses and propagates the input 310 to the second layer 340. Theinput 310 is further processed in the second layer 340 and propagated tothe third layer 360. The third layer 360 categorizes data to be providedto the output layer e80. More specifically, as shown in the example ofFIG. 4, a convolution 404 (e.g., a 5×5 convolution, etc.) is applied toa portion or window (also referred to as a “receptive field”) 402 of theinput 310 (e.g., a 32×32 data input, etc.) in the first layer 320 toprovide a feature map 406 (e.g., a (6×) 28×28 feature map, etc.). Theconvolution 404 maps the elements from the input 310 to the feature map406. The first layer 320 also provides subsampling (e.g., 2×2subsampling, etc.) to generate a reduced feature map 410 (e.g., a (6×)14×14 feature map, etc.). The feature map 410 undergoes a convolution412 and is propagated from the first layer 320 to the second layer 340,where the feature map 410 becomes an expanded feature map 414 (e.g., a(16×) 10×10 feature map, etc.). After subsampling 416 in the secondlayer 340, the feature map 414 becomes a reduced feature map 418 (e.g.,a (16×) 4×5 feature map, etc.). The feature map 418 undergoes aconvolution 420 and is propagated to the third layer 360, where thefeature map 418 becomes a classification layer 422 forming an outputlayer of N categories 424 with connection 426 to the convoluted layer422, for example.

FIG. 5 is a representation of an example implementation of an imageanalysis convolutional neural network 500. The convolutional neuralnetwork 500 receives an input image 502 and abstracts the image in aconvolution layer 504 to identify learned features 510-522. In a secondconvolution layer 530, the image is transformed into a plurality ofimages 530-538 in which the learned features 510-522 are eachaccentuated in a respective sub-image 530-538. The images 530-538 arefurther processed to focus on the features of interest 510-522 in images540-548. The resulting images 540-548 are then processed through apooling layer which reduces the size of the images 540-548 to isolateportions 550-554 of the images 540-548 including the features ofinterest 510-522. Outputs 550-554 of the convolutional neural network500 receive values from the last non-output layer and classify the imagebased on the data received from the last non-output layer. In certainexamples, the convolutional neural network 500 may contain manydifferent variations of convolution layers, pooling layers, learnedfeatures, and outputs, etc.

FIG. 6A illustrates an example configuration 600 to apply a learning(e.g., machine learning, deep learning, etc.) network to process and/orotherwise evaluate an image. Machine learning can be applied to avariety of processes including image acquisition, image reconstruction,image analysis/diagnosis, etc. As shown in the example configuration 600of FIG. 6A, raw data 610 (e.g., raw data 610 such as sonogram raw data,etc., obtained from an imaging scanner such as an x-ray, computedtomography, ultrasound, magnetic resonance, etc., scanner) is fed into alearning network 620. The learning network 620 processes the data 610 tocorrelate and/or otherwise combine the raw data 620 into processed data630 (e.g., a resulting image, etc.) (e.g., a “good quality” image and/orother image providing sufficient quality for diagnosis, etc.). Thelearning network 620 includes nodes and connections (e.g., pathways) toassociate raw data 610 with the processed data 630. The learning network620 can be a training network that learns the connections and processesfeedback to establish connections and identify patterns, for example.The learning network 620 can be a deployed network that is generatedfrom a training network and leverages the connections and patternsestablished in the training network to take the input raw data 610 andgenerate the resulting image 630, for example.

Once the learning 620 is trained and produces good images 630 from theraw image data 610, the network 620 can continue the “self-learning”process and refine its performance as it operates. For example, there is“redundancy” in the input data (raw data) 610 and redundancy in thenetwork 620, and the redundancy can be exploited.

If weights assigned to nodes in the learning network 620 are examined,there are likely many connections and nodes with very low weights. Thelow weights indicate that these connections and nodes contribute littleto the overall performance of the learning network 620. Thus, theseconnections and nodes are redundant. Such redundancy can be evaluated toreduce redundancy in the inputs (raw data) 610. Reducing input 610redundancy can result in savings in scanner hardware, reduced demands oncomponents, and also reduced exposure dose to the patient, for example.

In deployment, the configuration 600 forms a package 600 including aninput definition 610, a trained network 620, and an output definition630. The package 600 can be deployed and installed with respect toanother system, such as an imaging system, analysis engine, etc. Animage enhancer 625 can leverage and/or otherwise work with the learningnetwork 620 to process the raw data 610 and provide a result (e.g.,processed image data and/or other processed data 630, etc.). Thepathways and connections between nodes of the trained learning network620 enable the image enhancer 625 to process the raw data 610 to formthe image and/or other processed data result 630, for example.

As shown in the example of FIG. 6B, the learning network 620 can bechained and/or otherwise combined with a plurality of learning networks621-623 to form a larger learning network. The combination of networks620-623 can be used to further refine responses to inputs and/orallocate networks 620-623 to various aspects of a system, for example.

In some examples, in operation, “weak” connections and nodes caninitially be set to zero. The learning network 620 then processes itsnodes in a retaining process. In certain examples, the nodes andconnections that were set to zero are not allowed to change during theretraining. Given the redundancy present in the network 620, it ishighly likely that equally good images will be generated. As illustratedin FIG. 6B, after retraining, the learning network 620 becomes DLN 621.The learning network 621 is also examined to identify weak connectionsand nodes and set them to zero. This further retrained network islearning network 622. The example learning network 622 includes the“zeros” in learning network 621 and the new set of nodes andconnections. The learning network 622 continues to repeat the processinguntil a good image quality is reached at a learning network 623, whichis referred to as a “minimum viable net (MVN)”. The learning network 623is a MVN because if additional connections or nodes are attempted to beset to zero in learning network 623, image quality can suffer.

Once the MVN has been obtained with the learning network 623, “zero”regions (e.g., dark irregular regions in a graph) are mapped to theinput 610. Each dark zone is likely to map to one or a set of parametersin the input space. For example, one of the zero regions may be linkedto the number of views and number of channels in the raw data. Sinceredundancy in the network 623 corresponding to these parameters can bereduced, there is a highly likelihood that the input data can be reducedand generate equally good output. To reduce input data, new sets of rawdata that correspond to the reduced parameters are obtained and runthrough the learning network 621. The network 620-623 may or may not besimplified, but one or more of the learning networks 620-623 isprocessed until a “minimum viable input (MVI)” of raw data input 610 isreached. At the MVI, a further reduction in the input raw data 610 mayresult in reduced image 630 quality. The MVI can result in reducedcomplexity in data acquisition, less demand on system components,reduced stress on patients (e.g., less breath-hold or contrast), and/orreduced dose to patients, for example.

By forcing some of the connections and nodes in the learning networks620-623 to zero, the network 620-623 to build “collaterals” tocompensate. In the process, insight into the topology of the learningnetwork 620-623 is obtained. Note that network 621 and network 622, forexample, have different topology since some nodes and/or connectionshave been forced to zero. This process of effectively removingconnections and nodes from the network extends beyond “deep learning”and can be referred to as “deep-deep learning”, for example.

In certain examples, input data processing and deep learning stages canbe implemented as separate systems. However, as separate systems,neither module may be aware of a larger input feature evaluation loop toselect input parameters of interest/importance. Since input dataprocessing selection matters to produce high-quality outputs, feedbackfrom deep learning systems can be used to perform input parameterselection optimization or improvement via a model. Rather than scanningover an entire set of input parameters to create raw data (e.g., whichis brute force and can be expensive), a variation of active learning canbe implemented. Using this variation of active learning, a startingparameter space can be determined to produce desired or “best” resultsin a model. Parameter values can then be randomly decreased to generateraw inputs that decrease the quality of results while still maintainingan acceptable range or threshold of quality and reducing runtime byprocessing inputs that have little effect on the model's quality.

FIG. 7 illustrates example training and deployment phases of a learningnetwork, such as a deep learning or other machine learning network. Asshown in the example of FIG. 7, in the training phase, a set of inputs702 is provided to a network 704 for processing. In this example, theset of inputs 702 can include facial features of an image to beidentified. The network 704 processes the input 702 in a forwarddirection 706 to associate data elements and identify patterns. Thenetwork 704 determines that the input 702 represents a lung nodule 708.In training, the network result 708 is compared 710 to a known outcome712. In this example, the known outcome 712 is a frontal chest (e.g.,the input data set 702 represents a frontal chest identification, not alung nodule). Since the determination 708 of the network 704 does notmatch 710 the known outcome 712, an error 714 is generated. The error714 triggers an analysis of the known outcome 712 and associated data702 in reverse along a backward pass 716 through the network 704. Thus,the training network 704 learns from forward 706 and backward 716 passeswith data 702, 712 through the network 704.

Once the comparison of network output 708 to known output 712 matches710 according to a certain criterion or threshold (e.g., matches ntimes, matches greater than x percent, etc.), the training network 704can be used to generate a network for deployment with an externalsystem. Once deployed, a single input 720 is provided to a deployedlearning network 722 to generate an output 724. In this case, based onthe training network 704, the deployed network 722 determines that theinput 720 is an image of a frontal chest 724.

FIG. 8 illustrates an example product leveraging a trained networkpackage to provide a deep and/or other machine learning productoffering. As shown in the example of FIG. 8, an input 810 (e.g., rawdata) is provided for preprocessing 820. For example, the raw input data810 is preprocessed 820 to check format, completeness, etc. Once thedata 810 has been preprocessed 820, patches are created 830 of the data.For example, patches or portions or “chunks” of data are created 830with a certain size and format for processing. The patches are then fedinto a trained network 840 for processing. Based on learned patterns,nodes, and connections, the trained network 840 determines outputs basedon the input patches. The outputs are assembled 850 (e.g., combinedand/or otherwise grouped together to generate a usable output, etc.).The output is then displayed 860 and/or otherwise output to a user(e.g., a human user, a clinical system, an imaging modality, a datastorage (e.g., cloud storage, local storage, edge device, etc.), etc.).

As discussed above, learning networks can be packaged as devices fortraining, deployment, and application to a variety of systems. FIGS.9A-9C illustrate various learning device configurations. For example,FIG. 9A shows a general learning device 900. The example device 900includes an input definition 910, a learning network model 920, and anoutput definition 930. The input definition 910 can include one or moreinputs translating into one or more outputs 930 via the network 920.

FIG. 9B shows an example training device 901. That is, the trainingdevice 901 is an example of the device 900 configured as a traininglearning network device. In the example of FIG. 9B, a plurality oftraining inputs 911 are provided to a network 921 to develop connectionsin the network 921 and provide an output to be evaluated by an outputevaluator 931. Feedback is then provided by the output evaluator 931into the network 921 to further develop (e.g., train) the network 921.Additional input 911 can be provided to the network 921 until the outputevaluator 931 determines that the network 921 is trained (e.g., theoutput has satisfied a known correlation of input to output according toa certain threshold, margin of error, etc.).

FIG. 9C depicts an example deployed device 903. Once the training device901 has learned to a requisite level, the training device 901 can bedeployed for use. While the training device 901 processes multipleinputs to learn, the deployed device 903 processes a single input todetermine an output, for example. As shown in the example of FIG. 9C,the deployed device 903 includes an input definition 913, a trainednetwork 923, and an output definition 933. The trained network 923 canbe generated from the network 921 once the network 921 has beensufficiently trained, for example. The deployed device 903 receives asystem input 913 and processes the input 913 via the network 923 togenerate an output 933, which can then be used by a system with whichthe deployed device 903 has been associated, for example.

Example Image Processing Systems and Methods to Determine RadiologicalFindings

FIG. 10 illustrates an example image processing system or apparatus 1000including an imaging system 1010 having a processor 1020 to processimage data stored in a memory 1030. As shown in the example of FIG. 10,the example processor 1020 includes an image quality checker 1022, apre-processor 1024, a learning network 1026, and an image enhancer 1028providing information to an output 1030. Image data acquired from apatient by the imaging system 1010 can be stored in an image data store1035 of the memory 1030, and such data can be retrieved and processed bythe processor 1020.

Radiologist worklists are prioritized by putting stat images first,followed by images in order from oldest to newest, for example. Bypractice, most intensive care unit (ICU) chest x-rays are ordered asSTAT. Since so many images are ordered as STAT, a radiologist can beunaware, among all the STAT images, which ones are really the mostcritical. In a large US healthcare institution, for example, a STATx-ray order from the emergency room (ER) is typically prioritized to beread by radiologists first and is expected to be read/reported inapproximately one hour. Other STAT x-ray orders, such as those acquiredin the ICU, are typically prioritized next such that they may take twoto four hours to be read and reported. Standard x-ray orders aretypically expected to be read/reported within one radiologist shift(e.g., 6-8 hours, etc.).

Often, if there is an overnight radiologist (e.g., in larger healthcarefacilities, etc.), the overnight radiologist is dedicated to readingadvanced imaging exams (e.g., CT, MR, etc.), and only will read x-raysif there is a special request. Morning chest x-ray rounds commonly occurevery day in the ICU, very early in the morning (e.g., 5 am, etc.). Adaytime radiologist shift, however, may not start until 8 am. Then, theradiologist will sit and read through all the morning round images. Ifthere is a critical finding (e.g., a patient health result that warrantsimmediate action such as a tension pneumothorax, mispositioned tube,impending ruptured aneurysm, etc.), the radiologist may not find it forseveral hours after the image was taken. In certain examples, an urgentfinding, such as an aneurysm at risk of bursting imminently, etc., isdistinguished from a critical finding that is important but at risk ofhappening later in time such as a suspicious mass that could becomemalignant, etc.

Additionally, when a tube or line is placed within a patient, it isstandard practice to take an x-ray to verify correct placement of thetube or line. Due to the delay in radiologist read/reporting, clinicalcare teams (e.g., nurse, intensivists, etc.) may read the chest x-rayimage(s) themselves to determine if any intervention is needed (e.g.,medication changes to manage fluid in the lungs, adjustment of amisplaced line/tube, or confirmation of correctly place tube so they canturn on the breathing machine or feeding tube, etc.). Depending on theclinical care team's experience, skill, or attention to detail, they maymiss critical findings that compromise the patient's health by delayingdiagnosis, for example. When a radiologist finds a critical finding inan x-ray, the standard practice is for them to physically call theordering physician and discuss the finding. In some cases, the orderingphysician confirms they are aware and saw the issue themselves; in othercases, it is the first time they are hearing the news and will need toquickly intervene to help the patient.

Thus, to improve image availability, system flexibility, diagnosis time,reaction time for treatment, and the like, certain examples provide anon-device/point-of-care notification of clinical finding such as to tella clinical team at the point of care (e.g., at a patient's bedside,etc.) to review an image as the image has a high likelihood of includinga critical finding. For images with critical findings, when the image ispushed to storage such as a PACS, an HL7 message can also be sent to anassociated PACS/radiology information system (RIS) and/or DICOM tag,which indicates a critical finding. A hospital information system canthen create/configure rules to prioritize the radiologist worklist basedon this information, for example.

Turning to the example of FIG. 10, the image quality checker 1022processes the retrieved image data to evaluate the quality of the imagedata according to one or more image quality measures to help ensure thatthe image is of sufficient quality (e.g., good quality, other expectedquality, etc.) for automated (e.g., machine learning, deep learning,and/or other artificial intelligence, etc.) processing of the imagedata. Image data failing to pass a quality check with respect to one ormore image quality measures can be rejected as of insufficient quality,with a notification generated to alert a technologist and/or other userof the quality control failure. In certain examples, artificialintelligence (AI) can be applied to analyze the image data to evaluateimage quality.

By hosting an AI algorithm on the imaging device 1010, a “quality checkAI” algorithm can be executed before a “critical condition AI” to helpensure that the image is of good quality/expected quality for the“critical condition AI” to perform well. The “quality check AI” can beused on the device as an assistant to the technologist (“Tech”) such aswhen the tech performs Quality Assurance (QA)/Quality Check (QC)practices on the images they acquire. For example, after each image isacquired, the Tech may review the image to ensure proper patientpositioning, collimation, exposure/technique, no patient jewelry orclothing obstructions, no artifacts, etc. If the Tech believes the imageis of good quality, then the Tech will “accept” the image. However, ifthe image fails the QC check, the Tech can “reject” the image and“retake” the image (e.g., re-obtain the image data through a subsequentimage acquisition).

Depending on the Tech's experience and skill, the Tech may have adifferent tolerance for accept/reject image quality. However, using AIembedded in the device 1010 allows the device 1010 processor 1020 toevaluate and notify the Tech if the image fails the “quality check AI”.The image fails the quality check AI, for example, if the image is oftoo poor quality to reliably run through a “critical condition AI”algorithm, but simultaneously, also indicating to the Tech that perhapsthe image should fail their manual/traditional QC activity as well, andthat the Tech should consider a “retake”. Thus, the image qualitychecker 1022 can provide feedback in real-time (or substantiallyreal-time given image data processing, transmission, and/orstorage/retrieval latency) such as at the patient bedside via the output1030 of the mobile x-ray system 200, 1010 indicating/recommending thatan image should be re-acquired, for example.

Thus, rather than relying on a Tech's manual assessment, the qualitychecker 1022 can leverage AI and/or other processing to analyze imageanatomy, orientation/position, sufficient contrast, appropriate dose,too much noise/artifacts, etc., to evaluate image quality andsufficiency to enable further automated analysis.

If image quality is sufficient and/or otherwise appropriate (e.g.,correct view/position, correct anatomy, acceptable contrast and/or noiselevel, etc.) for analysis, then the pre-processor 1024 processes theimage data and prepares the image data for clinical analysis. Forexample, the image data can be conditioned for processing by machinelearning, such as a deep learning network, etc., to identify one or morefeatures of interest in the image data. The pre-processor 1024 can applytechniques such as image segmentation to identify and divide differentregions or areas in the image, for example. The pre-processor 1024 canapply techniques such as cropping to select a certain region of interestin the image for further processing and analysis, for example. Thepre-processor 1024 can apply techniques such as down-sampling to scaleor reduce image data size for further processing (e.g., by presentingthe learning network 1026 with fewer samples representing the imagedata, etc.), for example.

The pre-processed image data is provided to the learning network 1026for processing of the image data to identify one or moreclinical/critical findings. As discussed above, the learning network1026, such as a deep learning network, other CNN, and/or other machinelearning network, etc., receives the pre-processed image data at itsinput nodes and evaluates the image data according to the nodes andconnective pathways of the learning network 1026 to correlate featuresidentified in the pre-processed image data with critical and/or otherclinical findings. Based on image intensity values, reference coordinateposition, proximity, and/or other characteristics, items determined inthe image data can be correlated with likely critical and/or otherclinical findings such as a severe pneumothorax, tube within the rightmainstem, free air in the bowel, etc.

For example, a large, highly curated set of X-Ray images can be used totrain a deep convolution network (e.g., the example network of FIGS.3-5, etc.) including several layers in an offline compute-intensiveenvironment. The network is trained to output classification labelsdepicting a detected pathology and are able to extract features that canlocalize and bound regions interest to the detected pathology. Thespecialized network is developed and trained to output quantificationmetrics such as fluid density, opacity and volumetric measurements, etc.As shown in the example of FIGS. 6A-9C, trained model(s) are deployedonto an X-Ray device (e.g., the imaging device 10, 200, 1010, etc.)which is either mobile or installed in a fixed X-Ray room. The processor1020 leverages the trained, deployed model(s) to infer properties,features, and/or other aspects of the image data by inputting the X-Rayimage into the trained network model(s). The deployed model(s) helpcheck quality and suitability of the image for inference via the imagequality checker 1022 and infer findings via the learning network 1026,for example. The images can be pre-processed in real time based onacquisition conditions that generated the image to improve accuracy andefficacy of the inference process. In certain examples, the learningnetwork(s) 1026 are trained, updated, and redeployed continuously and/orperiodically upon acquisition of additional curated data. As a result,more accurate and feature enhanced networks are deployed on the imagingdevice 1010.

In certain examples, a probability and/or confidence indicator or scorecan be associated with the indication of critical and/or other clinicalfinding(s), a confidence associated with the finding, a location of thefinding, a severity of the finding, a size of the finding, and/or anappearance of the finding in conjunction with another finding or in theabsence of another finding, etc. For example, a strength of correlationor connection in the learning network 1026 can translate into apercentage or numerical score indicating a probability of correctdetection/diagnosis of the finding in the image data, a confidence inthe identification of the finding, etc.

The image data and associated finding(s) can be provided via the output1030 to be displayed, reported, logged, and/or otherwise used in anotification or alert to a healthcare practitioner such as a Tech,nurse, intensivist, trauma surgeon, etc., to act quickly on the criticaland/or other clinical finding. In some examples, the probability and/orconfidence score, and/or a criticality index/score associated with thetype of finding, size of finding, location of finding, etc., can be usedto determine a severity, degree, and/or other escalation of thealert/notification to the healthcare provider. For example, certaindetected conditions result in a text-based alert to a provider to promptthe provider for closer review. Other, more serious conditions result inan audible and/or visual alert to one or more providers for moreimmediate action. Alert(s) and/or other notification(s) can escalate inproportion to an immediacy and/or other severity of a probable detectedcondition, for example.

Image data and associated finding(s) can be provided to image enhancer1028 for image post-processing to enhance the image data. For example,the image enhancer 1028 can process the image data based on thefinding(s) to accentuate the finding(s) in a resulting image. Thus, whenthe enhanced image data is provided to the output 1030 for display(e.g., via one or more devices such as a mobile device 1040, display1042, PACS and/or other information system 1044, etc.), the finding(s)are emphasized, highlighted, noted, and/or otherwise enhanced in theresulting displayed image, for example.

By running AI on the imaging device 1010, AI findings can be leveragedto conduct enhanced image processing. For example, if the AI detectstubes/lines present in the image data, then the device software canprocess the image using an image processing technique best for viewingtubes/lines. For example, tubes and/or other lines (e.g., catheter,feeding tube, nasogastric (NG) tube, endotracheal (ET) tube, chest tube,pacemaker leads, etc.) can be emphasized or enhanced in the image datathrough an image processing algorithm that decomposes the image datainto a set of spatial frequency bands. Non-linear functions can beapplied to the frequency bands to enhance contrast and reduce noise ineach band. Spatial frequencies including tubes and lines are enhancedwhile spatial frequencies including noise are suppressed. As a result,the tubes and lines are more pronounced in a resulting image. Similarly,a pneumothorax (e.g., an abnormal collection of air in pleural spacebetween a lung and the chest), fracture, other foreign object, etc.,representing a finding can be emphasized and/or otherwise enhanced in aresulting image, for example.

The enhanced image data and associated finding(s) can be output fordisplay, storage, referral, further processing, provision to acomputer-aided diagnosis (CAD) system, etc., via the output 1030. Theoutput 1030 can provide information to a plurality of connected devices1040-1044 for review, storage, relay, and/or further action, forexample.

The more contextual information available about a patient, the moreinformed, accurate, and timely diagnosis a physician can make.Similarly, the more information provided to an AI algorithm model, themore accurate the prediction generated by the model. As described above,AI models can be used to deploy algorithms on an imaging device toprovide bedside, real-time point of care notifications when a patienthas a critical finding, without suffering form latency or connectivityrisks from running an AI model remotely. However, contextual data is notreadily available on the imaging device, so certain examples retrieveand use contextual patient data for AI algorithm input on the imagingdevice.

Contextual patient information can be used to improve accuracy of anartificial intelligence algorithm model, for example. For example, apneumothorax, or collapsed lung, is more common in tall, thin men andpeople with a prior history of the condition. Therefore, a chest x-raypneumothorax AI detection algorithm, when provided electronic medicalrecord information regarding patient body type and patient medicalhistory, can more accurately predict the presence of the pneumothoraxdisease.

Additionally, contextual patient information can be used to determinewhether a clinical condition is worsening, improving, or staying thesame over time, for example. For example, a critical test result from achest x-ray exam is considered to be a “new or significant progressionof pneumothorax”, in which the radiologist shall call the orderingpractitioner and verbally discuss the findings. Although, chest x-raypneumothorax AI detection algorithm that resides on the mobile x-raysystem, may not have the patient's prior chest x-ray to determinewhether a pneumothorax finding is new or significantly progressed.Therefore, providing the AI algorithm with prior imaging exams, would benecessary to determine whether the pneumothorax finding shall beconsidered critical or not.

Some examples of contextual patient data, which can be used byartificial intelligence for detection, classification, segmentation,etc., include: electronic medical record data (e.g., age, gender,weight, height, body mass index (BMI), smoking status, medication list,existing conditions, prior conditions, etc.), prior images (e.g., x-ray,CT, MR, etc.), electrocardiogram (EKG) heart monitor data, patienttemperature, oxygen saturation/O2 meter data, blood pressureinformation, ventilator metrics (e.g., respiratory rate, etc.), fluidand medication administration data (e.g., intravenous (IV) fluidadministration, etc.), etc. For example, electronic medical record datacan be retrieved using an HL7 query retrieve message, prior images canbe retrieved using a PACS query, data from wireless patient monitoringdevices (e.g., an EKG heart monitor, etc.) can be retrieved usingwireless connectivity (e.g., Wi-Fi, etc.) between devices, etc.

Thus, contextual patient data can be collected on an imaging device toenable more accurate and advanced artificial intelligence algorithms. Incertain examples, change versus no change “AI algorithms can be modeledon an imaging device to detect progression of disease. Leveragingmore/diverse data sources allows the system to create higher-performingAI algorithms, for example. In certain examples, an AI algorithm modelcan be implemented on the imaging device, on an edge server, and/or on acloud-based server where other contextual data is collected, forexample.

While certain examples generate and apply an AI algorithm model using acurrent imaging exam as input, other examples pull additional data fromsources off the imaging device to form input for an AI model.

While example implementations are illustrated in conjunction with FIGS.1-10, elements, processes and/or devices illustrated in conjunction withFIGS. 1-10 can be combined, divided, re-arranged, omitted, eliminatedand/or implemented in any other way. Further, components disclosed anddescribed herein can be implemented by hardware, machine readableinstructions, software, firmware and/or any combination of hardware,machine readable instructions, software and/or firmware. Thus, forexample, components disclosed and described herein can be implemented byanalog and/or digital circuit(s), logic circuit(s), programmableprocessor(s), application specific integrated circuit(s) (ASIC(s)),programmable logic device(s) (PLD(s)) and/or field programmable logicdevice(s) (FPLD(s)). When reading any of the apparatus or system claimsof this patent to cover a purely software and/or firmwareimplementation, at least one of the components is/are hereby expresslydefined to include a tangible computer readable storage device orstorage disk such as a memory, a digital versatile disk (DVD), a compactdisk (CD), a Blu-ray disk, etc. storing the software and/or firmware.

Flowcharts representative of example machine readable instructions forimplementing components disclosed and described herein are shown inconjunction with at least FIGS. 11-12 and 28-29. In the examples, themachine readable instructions include a program for execution by aprocessor such as the processor 3012 shown in the example processorplatform 3000 discussed below in connection with FIG. 30. The programmay be embodied in machine readable instructions stored on a tangiblecomputer readable storage medium such as a CD-ROM, a floppy disk, a harddrive, a digital versatile disk (DVD), a Blu-ray disk, or a memoryassociated with the processor 3012, but the entire program and/or partsthereof could alternatively be executed by a device other than theprocessor 3012 and/or embodied in firmware or dedicated hardware.Further, although the example program is described with reference to theflowcharts illustrated in conjunction with at least FIGS. 11-12 and28-29, many other methods of implementing the components disclosed anddescribed herein may alternatively be used. For example, the order ofexecution of the blocks may be changed, and/or some of the blocksdescribed may be changed, eliminated, or combined. Although theflowcharts of at least FIGS. 11-12 and 28-29 depict example operationsin an illustrated order, these operations are not exhaustive and are notlimited to the illustrated order. In addition, various changes andmodifications may be made by one skilled in the art within the spiritand scope of the disclosure.

For example, blocks illustrated in the flowchart may be performed in analternative order or may be performed in parallel.

As mentioned above, the example processes of at least FIGS. 11-12 and28-29 may be implemented using coded instructions (e.g., computer and/ormachine readable instructions) stored on a tangible computer readablestorage medium such as a hard disk drive, a flash memory, a read-onlymemory (ROM), a compact disk (CD), a digital versatile disk (DVD), acache, a random-access memory (RAM) and/or any other storage device orstorage disk in which information is stored for any duration (e.g., forextended time periods, permanently, for brief instances, for temporarilybuffering, and/or for caching of the information). As used herein, theterm tangible computer readable storage medium is expressly defined toinclude any type of computer readable storage device and/or storage diskand to exclude propagating signals and to exclude transmission media. Asused herein, “tangible computer readable storage medium” and “tangiblemachine readable storage medium” are used interchangeably. Additionallyor alternatively, the example processes of at least FIGS. 11-12 and28-29 can be implemented using coded instructions (e.g., computer and/ormachine readable instructions) stored on a non-transitory computerand/or machine readable medium such as a hard disk drive, a flashmemory, a read-only memory, a compact disk, a digital versatile disk, acache, a random-access memory and/or any other storage device or storagedisk in which information is stored for any duration (e.g., for extendedtime periods, permanently, for brief instances, for temporarilybuffering, and/or for caching of the information). As used herein, theterm non-transitory computer readable medium is expressly defined toinclude any type of computer readable storage device and/or storage diskand to exclude propagating signals and to exclude transmission media. Asused herein, when the phrase “at least” is used as the transition termin a preamble of a claim, it is open-ended in the same manner as theterm “comprising” is open ended. In addition, the term “including” isopen-ended in the same manner as the term “comprising” is open-ended.

As shown in the example method 1100 depicted in FIG. 11, acquired imagedata can be analyzed by the imaging device 1010 at the location of imageacquisition (e.g., patent bedside, imaging room, etc.) to evaluate imagequality and identify likely critical and/or other clinical findings totrigger further image and patient review and action. At block 1110,acquired image data acquired by the device 1010 is processed by thedevice 1010 to evaluate the quality of the image data to help ensurethat the image is of sufficient quality (e.g., good quality, otherexpected quality, etc.) for automated (e.g., machine learning, deeplearning, and/or other artificial intelligence, etc.) processing of theimage data. Image data failing to pass a quality check can be rejectedas of insufficient quality, with a notification generated to alert atechnologist and/or other user of the quality control failure. Incertain examples, artificial intelligence (AI) can be applied by theimage quality checker 1022 to analyze the image data to evaluate imagequality.

By hosting an AI algorithm on the imaging device 1010, a “quality checkAI” algorithm can be executed before a “critical condition AI” to helpensure that the image is of good quality/expected quality for the“critical condition AI” to perform well. The “quality check AI” can beused on the device as an assistant to the technologist (“Tech”) such aswhen the tech performs Quality Assurance (QA)/Quality Check (QC)practices on the images they acquire. Using AI embedded in the device1010 allows the device 1010 processor 1020 to evaluate and notify 1115the Tech if the image fails the “quality check Al”. The image fails thequality check AI, for example, if the image is of too poor quality toreliably run through a “critical condition AI” algorithm, butsimultaneously, also indicating to the Tech that perhaps the imageshould fail their manual/traditional QC activity as well, and that theTech should consider a “retake”. Thus, the image quality checker 1022can provide feedback in real-time (or substantially real-time givenimage data processing, transmission, and/or storage/retrieval latency)such as at the patient bedside via the output 1030 of the mobile x-raysystem 200, 1010 indicating/recommending via a notification 1115 that animage should be re-acquired, for example. For example, the notification1115 can be provided via an overlay on the mobile device 1040, display1042, etc., to show localization (e.g., via a heatmap, etc.) of the AIfinding and/or associated information.

Thus, rather than relying on a Tech's manual assessment, the qualitychecker 1022 can leverage AI and/or other processing to analyze imageanatomy, orientation/position, sufficient contrast, appropriate dose,too much noise/artifacts, etc., to evaluate image quality andsufficiency to enable further automated analysis.

If image quality is sufficient and/or otherwise appropriate (e.g.,correct view/position, correct anatomy, acceptable patient positioning,contrast and/or noise level, etc.) for analysis, then, at block 1120,the image data is pre-processed to prepare the image data for clinicalanalysis. For example, the image data can be conditioned for processingby machine learning, such as a deep learning network, etc., to identifyone or more features of interest in the image data. The pre-processor1024 can apply techniques such as image segmentation to identify anddivide different regions or areas in the image, for example. Thepre-processor 1024 can apply techniques such as cropping to select acertain region of interest in the image for further processing andanalysis, for example. The pre-processor 1024 can apply techniques suchas down-sampling, anatomical segmentation, normalizing with mean and/orstandard deviation of training population, contrast enhancement, etc.,to scale or reduce image data size for further processing (e.g., bypresenting the learning network 1026 with fewer samples representing theimage data, etc.), for example.

At block 1130, the pre-processed image data is provided to the learningnetwork 1026 for processing of the image data to identify one or moreclinical/critical findings. As discussed above, the learning network1026, such as a deep learning network, other CNN and/or other machinelearning network, etc., receives the pre-processed image data at itsinput nodes and evaluates the image data according to the nodes andconnective pathways of the learning network 1026 to correlate featuresidentified in the pre-processed image data with critical and/or otherclinical findings. Based on image intensity values, reference coordinateposition, proximity, and/or other characteristics, items determined inthe image data can be correlated with likely critical and/or otherclinical findings such as a severe pneumothorax, tube within the rightmainstem, free air in the bowel, etc.

For example, a large, highly curated set of X-Ray images can be used totrain a deep convolution network (e.g., the example network of FIGS.3-5, etc.) including several layers in an offline compute-intensiveenvironment. The network is trained to output classification labelsdepicting a detected pathology and are able to extract features that canlocalize and bound regions interest to the detected pathology. Thespecialized network is developed and trained to output quantificationmetrics such as fluid density, opacity and volumetric measurements, etc.As shown in the example of FIGS. 6A-9C, trained model(s) are deployedonto an X-Ray device (e.g., the imaging device 10, 200, 1010, etc.)which is either mobile or installed in a fixed X-Ray room. The processor1020 leverages the trained, deployed model(s) to infer properties,features, and/or other aspects of the image data by inputting the X-Rayimage into the trained network model(s). The deployed model(s) helpcheck quality and suitability of the image for inference via the imagequality checker 1022 and infer findings via the learning network 1026,for example. The images can be pre-processed in real time based onacquisition conditions that generated the image to improve accuracy andefficacy of the inference process. In certain examples, the learningnetwork(s) 1026 are trained, updated, and redeployed continuously and/orperiodically upon acquisition of additional curated data. As a result,more accurate and feature enhanced networks are deployed on the imagingdevice 1010.

In certain examples, a probability and/or confidence indicator or scorecan be associated with the indication of critical and/or other clinicalfinding(s), as well as a size of the finding, location of the finding,severity of the finding, etc. For example, a strength of correlation orconnection in the learning network 1026 can translate into a percentageor numerical score indicating a probability of correctdetection/diagnosis of the finding in the image data, a confidence inthe identification of the finding, etc.

The image data and associated finding(s) can be provided via the output1030 to be displayed, reported, logged, and/or otherwise used in anotification or alert 1135 to a healthcare practitioner such as a Tech,nurse, intensivist, trauma surgeon, and/or clinical system, etc., to actquickly on the critical and/or other clinical finding. In some examples,the probability and/or confidence score, and/or a criticalityindex/score associated with the type of finding, can be used todetermine a severity, degree, and/or other escalation of thealert/notification to the healthcare provider. For example, certaindetected conditions result in a text-based alert to a provider to promptthe provider for closer review. Other, more serious conditions result inan audible and/or visual alert to one or more providers for moreimmediate action. Alert(s) and/or other notification(s) can escalate inproportion to an immediacy and/or other severity of a probable detectedcondition, for example.

At block 1140, image data is enhanced based on associated finding(s)identified by the learning network 1026. For example, the image enhancer1028 can process the image data based on the finding(s) to accentuatethe finding(s) in a resulting image. Thus, when the enhanced image datais provided to the output 1030 for display (e.g., via one or moredevices such as a mobile device 1040, display 1042, PACS and/or otherinformation system 1044, etc.), the finding(s) are emphasized,highlighted, noted, and/or otherwise enhanced in the resulting displayedimage, for example.

By running AI on the imaging device 1010, AI findings can be leveragedto conduct enhanced image processing. For example, if the AI detectstubes/lines present in the image data, then the device software canprocess the image using an image processing technique best for viewingtubes/lines.

The enhanced image data and associated finding(s) can be output fordisplay, storage, referral, further processing, provision to acomputer-aided diagnosis (CAD) system, etc., via the output 1030. Theoutput 1030 can provide information to a plurality of connected devices1040-1044 for review, storage, relay, and/or further action, forexample. As shown in the example of FIG. 11, enhanced image data andassociated finding(s) can be output for display on a device 1150 (e.g.,a handheld or mobile device, etc.), displayed on a workstation 1152(e.g., an information system, a display associated with the imagingdevice 1010, etc.), and/or sent to a clinical information system such asa PACS, RIS, enterprise archive, etc., for storage and/or furtherprocessing 1154.

FIG. 12 illustrates a flow diagram for example implementation ofchecking image quality (1110) and applying artificial intelligence(1130) to image data to determine critical and/or other clinicalfindings in the image data.

Portable, real-time, at point of patient care, at point of imageacquisition, dynamic determination and prompting for further action,integrated into imaging device. At 1202, image data, such as DICOM imagedata, is provided from a mobile x-ray imaging device (e.g., the device200 and/or 1010, etc.). At 1204, metadata associated with the image data(e.g., DICOM header information, other associated metadata, etc.) isanalyzed to determine whether the image data matches a position andregion indicated by the metadata. For example, if the DICOM metadataindicates that the image is a frontal (e.g., anteroposterior (AP) orposteroanterior (PA)) chest image, then an analysis of the image datashould confirm that position (e.g., location and orientation, etc.). Ifthe image does not match its indicated position and region, then, at1206, a notification, alert, and/or warning is generated indicating thatthe image is potentially improper. The warning can be an audible,visual, and/or system alert or other notification, for example, and canprompt a user for further action (e.g., re-acquire the image data,etc.), trigger a system for further action, log the potential error,etc.

If the image data appears to match its prescribed position and region,then, at 1208, the image data is analyzed to determine whether the imagepasses image quality control check(s). For example, the image data isanalyzed to determine whether the associated image has good patientpositioning (e.g., the patient is positioned such that an anatomy orregion of interested is centered in the image, etc.). Other qualitycontrol checks can include an evaluation of sufficient contrast, ananalysis of a level of noise or artifact in the image, an examination ofappropriate/sufficient dosage for image clarity, etc.

If the image fails a quality control check, then, at 1210, a warning ofcompromised image quality is generated. For example, a user, othersystem, etc., can receive an alert and/or other notification (e.g., avisual and/or audible alert on screen, via message, log notation,trigger, etc.) that the image quality may not be sufficient and/or maypresent issues in evaluating the image data to determine clinicalfinding(s). At 1212, settings and/or other input is evaluated todetermine whether to proceed with further image processing. For example,user input in response to the notification can indicate whether or notto proceed anyway, and/or a configuration setting, etc., can specify adefault instruction or threshold regarding whether or not to proceedwith further image analysis despite image quality concerns. If theinstruction is not to proceed, then the process 1200 ends.

If analysis is to proceed (e.g., because the image passes qualitycheck(s) and/or an instruction indicates to proceed despite imagequality concerns, etc.), then, at 1214, the image data is evaluated withrespect to a clinical check. For example, a deep learning network,machine learning, and/or other AI is applied to analyze the image datato detect the presence of a critical and/or other clinical finding. Forexample, image data can be processed by the learning network 1026 toidentify a severe pneumothorax and/or other condition (e.g., tube withinthe right mainstem, free air in the bowel, fracture, tumor, lesion,other foreign object, etc.) in the image data. If no finding isdetermined, then the process 1200 ends.

If, however, a finding is determined, then, at 1216, a finding alertand/or other notification is generated. For example, a critical findingalert is generated based on the identification of a pneumothorax,incorrect position of an ET tube, position of tube in right main stem,etc. The alert can be generated in proportion to and/or othercorrelation with a severity/urgency of the clinical finding, confidencein the finding, type of finding, location of the finding, and/orappearance of the finding in conjunction with another finding or inabsence of another finding, for example. For example, a critical findingcan be alerted more urgently to a healthcare practitioner and/or otheruser than a less-critical clinical finding. On-screen alert(s) can be13-displayed, HL7 messages can be provided to the RIS, etc. In certainexamples, image data can be re-processed such as by the image enhancer1028 to more optimally display the finding(s) to a user.

FIGS. 13-20 illustrate example displays to provide output and facilitateinteraction including on-screen alerts, indicators, notifications, etc.,in accordance with the apparatus, systems, and methods described abovein connection with FIGS. 1-12. The example displays can be provided viathe imaging devices 10, 200, 1010, and/or a separate handheld or mobilecomputing device, workstation, etc.

FIG. 13 shows an example graphical user interface (GUI) 1300 including adeviation index (DI) 1310 (e.g., an indication of correct imageacquisition technique with 0.0 being a perfect exposure) and anindication of priority 1320 (e.g., from processing by the system 1000including AI). As shown in the example of FIG. 13, the priorityindication 1320 is high 1322. FIG. 14 illustrates the example GUI 1300with a priority indication 1320 of medium 1324. FIG. 15 illustrates theexample GUI 1300 with a priority indication 1310 of low 1326.

FIG. 16 illustrates an example GUI 1600 including a DI 1610, a qualitycontrol indicator 1620 (e.g., pass or fail for acceptable quality,etc.), a criticality index 1630 (e.g., normal, abnormal, critical,etc.), a criticality value 1635 associated with the criticality index1630, an indication of finding 1640 (e.g., mass, fracture, pneumothorax,etc.), and an indication of size or severity 1650 (e.g., small, medium,large, etc.). Thus, a user can interact with the example GUI 1600 andevaluate the DI 1610, quality indication 1620, criticality range 1630and value 1635 for clinical impression, type of finding 1640, andseverity of finding 1650.

FIG. 17 illustrates another example GUI 1700, similar to but reducedfrom the example GUI 1600, including a DI 1710, a criticality impression1720, an indication of finding 1730, and an indication of severity 1740.FIG. 18 illustrates an example GUI 1800 similar to the example GUI 1700further including an overlay 1810 of the finding on the image.

FIG. 19 illustrates an example GUI 1900 providing potential findingsfrom AI in a window 1910 overlaid on the image viewer display of the GUI1900. FIG. 20 illustrates another view of the example GUI 1900 in whichentries 2002, 2004 in the AI findings window 1910 have been expanded toreveal further information regarding the respective finding 2002, 2004.FIG. 21 illustrates a further view of the example GUI 1900 in which theAI findings window 1910 has been reduced to a miniature representation2110, selectable to view information regarding the findings.

FIG. 22 illustrates an example GUI 2200 in which a finding 2210 ishighlighted on an associated image, and related information 2220 is alsodisplayed. FIG. 23 illustrates an example configuration interface 2300to configure AI to process image and/or other data and generatefindings.

FIG. 24 illustrates a first example abbreviated GUI 2400 (e.g., aweb-based GUI, etc.) displayable on a smartphone 2410 and/or othercomputing device, and a second abbreviated GUI 2420 shown on a tablet2430. As shown in the example of FIG. 24, the tablet 2430 can be mountedwith respect to the imaging device 10, 200, 1010 for viewing andinteraction by an x-ray technician and/or other healthcare practitioner,for example.

In certain examples, an exam can be prioritized on a worklist based onthe evaluation of the exam and detection of a critical finding. Forexample, a message can be sent from the imaging modality (e.g., a GEOptima™ XR240 x-ray system, GE LOGIQ™ ultrasound system, other mobileimaging system, etc.) to an information system (e.g., a RIS, PACS, etc.)to elevate an image on the worklist when a critical finding is detected.

FIG. 25 illustrates an example system configuration 2500 in which theimaging system 1010 interfaces with a broker device 2510 to communicatewith the PACS 1044, a RIS 2520, and/or other health information system,for example. As shown in the example of FIG. 25, the RIS 2520 provides amodality worklist (MWL) to the imaging device 1010. The MWL can beprovided as a service-object pair (SOP), for example, to enable theimaging system 1010 to query for patient demographics and study detailsfrom an MWL service class provider, such as the RIS 2520. For example,the imaging system 1010 can query the RIS 2520 for a list of patientssatisfying a criterion(-ia), and the RIS 2520 responds with results.

As shown in the example of FIG. 25, the RIS 2520 provides the MWL SOP tothe imaging device 1010, and the broker 2510 (e.g., an AI/HL7/DICOMbroker, etc.) facilitates a query by the imaging device 1010 andresponse by the RIS 2520. The imaging device 1010 can also use thebroker 2510 to send a message to the MS 2520 to move up an exam on theworklist when a critical finding is detected, which results in anupdated MWL provided from the MS 2520 to the imaging device 1010.Messages can be exchanged via the broker 2510 as a cross-platform HL7and/or DICOM interface engine sending bi-directional HL7/DICOM messagesbetween systems and/or applications running on the systems over multipletransports, for example.

Output from the imaging device 1010 can be stored on the PACS 1044and/or other device (e.g., an enterprise archive, vendor neutralarchive, other data store, etc.). For example, a DICOM storage SOP canbe used to transfer images, alerts, and/or other data from the imagingsystem 1010 to the PACS 1044.

Thus, rules can be created to determine image/exam priority, and thoserules can be stored such as in a DICOM header of an image sent to thePACS 1044. An AI model can be used to set a score or a flag in the DICOMheader (e.g., tag the DICOM header) to be used a rule to prioritizethose exams. Thus, a DICOM header tag (e.g., reflecting the score orflag, etc.) can be used to build priority rules. For example, a flag canbe used to indicate an urgent or STAT exam to be reviewed. A score canbe used to assign a relative degree of priority, for example. HL7messages can be communicated to and from the imaging device 1010 via thebroker 2510 to provide prioritization instructions, as well as otherstructured reports, DICOM data, HL7 messages, etc. Using DICOM headerand HL7 information, a client system, such as the RIS 2520, PACS 1044,etc., can determine priority.

In certain examples, prioritization rules can be made available on acloud-based server, an edge device, and/or a local server to enablecross-modality prioritization of exams in a worklist. Thus, rather thanor in addition to prioritizing based on wait time, physician, cost,etc., AI processing of image data can influence and/or dictate examand/or follow-up priority, and the prioritization scheme can bedistributed within and/or across, modalities, locations, etc., toimprove outcomes, for example. AI processing provides instantnotification at the imaging device 1010 as well as longer-termprioritization determining an ordering of images, exams, and/or otherdata for review.

FIG. 26 illustrates an example system configuration 2600 in which anartificial intelligent model executes on an edge device 2610 to providepoint of care alerts on a vendor neutral imaging machine 1010. The edgedevice 2610 can interface between local systems and a cloud-basedlearning factory 2630, for example. The edge device 2610 (e.g., a tabletand/or other handheld computing device, laptop, etc.) executes an AImodel and receives DICOM images from the imaging system 1010 as well asradiology reading reports from the RIS 2520. The edge device 2610 postsdata to the cloud factory 2630 including image data and associatedreport(s) for AI processing, data aggregation on the cloud, etc.

As shown in the example of FIG. 26, the imaging system 1010 can alsosend images to the PACS 1044 for storage. The PACS 1044 and the MS 2520can interact to exchange information such as provide images to the MS2520 to allow a user at a workstation 2640 to read the image and dictatea report.

As shown in the example of FIG. 26, an AI model executing on the edgedevice 2610 can generate a real-time AI report 2650 for a user, othersystem, application, etc. Thus, the edge device 2610 can providereal-time alerts at the point of care to trigger follow-up action, forexample.

In certain examples, such as shown in FIG. 26, a workstation 2660associated with the cloud-based learning factory 2630 can receiveimages, reports, etc., to review and evaluate the AI's assessment of theimages, etc. Feedback can be used to adjust the AI models, andre-annotate images for reporting, follow-up, etc.

Thus, the edge device 2610 can be positioned near the imaging device1010 and be mobile to provide AI image processing and critical findingfeedback, alerting, etc., while serving as an intermediary between localsystems 1010, 1044, 2520, and a remote cloud-based system 2630. Thepower of the cloud-based learning factory 2630 can be used to bolsterlocal on-device 2610 AI capabilities and deploy updated AI models to theedge device 2610 to improve processing of the data, for example.

In certain examples, results from previous-inferenced image(s) can beprovided via the edge device 2610 to generate an AI point of care alertbased on a delta or difference in inference results from currentsresults to the prior results. Thus, an evolution or change in resultscan be evaluated and used to trigger (or withdraw) a point of carealert. The edge device 2610 can retrieve prior and/or other analysisfrom the health cloud 2630, for example.

FIG. 27 illustrates an example system 2700 to incorporate and compare AIresults between current and prior exams. As shown in the example of FIG.27, an AI container and/or other virtual machine (e.g., a Dockercontainer, etc.) 2710 instantiates an AI inferencing engine whichproduces AI results and provides the AI results (e.g., via JSON, etc.)to form a context 2720 (e.g., an AI-augmented patient and/or exposurecontext, etc.). The context 2720 forms enriched AI results to provide tothe broker 2510, which conveys those results to connected systems suchas the RIS 2520, PACS 1044, etc. The broker 2510 also processes the AIresults and facilitates aggregation and querying of AI results for an AIresults comparator 2740, which also receives AI results from the AIcontainer 2710, for example.

AI results can be queried via the broker 2510 based on patientidentifier (ID), exam type, imaging device, etc. The comparator 2740generates a change notification 2750 when current AI results havediverged and/or otherwise differ from prior AI results of the image datafor the patient, for example. The edge device 2610, cloud system 2630,and/or other recipient can receive the change notification 2750 totrigger a point of care alert and/or additional actions to follow-up onthe identified change, for example.

As shown in the example of FIG. 27, the broker 2510 can include an orderupdate channel 2732 to update an order with respect to a patient at theRIS 2520 and/or the PACS 10144, and a database update channel 2734. AIresults can be provided to the database update channel 2734 to update anAI database 2736, for example. A database read channel 2738 in thebroker 2510 can be used to query AI results from the data store 2736 forthe comparator 2740, for example.

FIG. 28 illustrates a flow diagram for a method 2800 to prioritize, in aworklist, an exam related to a critical finding for review. At block2802, a critical finding is detected in image data at a modality 1010.For example, an AI model running at the imaging device 1010 identifies acritical finding (e.g., presence of a pneumothorax, lesion, fracture,etc.) prompting further review. At block 2804, the image data is stored.For example, the image data related to the critical finding is stored atthe PACS 1044. At block 2806, a message is sent from the modality 1010to the RIS 2520 to adjust the worklist based on the critical finding.Thus, the imaging device 1010 can instruct the RIS 2520 to move up anexam due to the identification of a critical finding.

FIG. 29 illustrates a flow diagram of a method 2900 to compare currentand prior AI analyses of image data to generate a notification for apoint of care alert. At block 2902, prior AI image processing resultsare received (e.g., at the AI Container 2710, the broker 2510, etc.). Atblock 2904, enriched AI results are generated with patient and/orexposure context. Thus, the context 2720 enriches the result data withinformation regarding the patient, the image exposure, the condition,history, environment, etc.

At block 2906, connected systems are updated based on the enriched AIresults via the broker 2510. For example, an exam order associated withthe patient can be updated at the RIS 2520 based on the enriched AIresults. Additionally, at block 2908, a database 2734 of AI resultinformation can be updated. At block 2910, query results are provided tothe comparator 2740, which, at block 2912, compares current AI resultswith prior AI results for the patient to determine a difference betweenthe results. Thus, the comparator 2740 can detect a change in the AIanalysis of the patient's image data. In certain examples, thecomparator 2740 can indicate a direction of change, a trend in thechange, etc.

At block 2914, a change notification is generated by the comparator 2740when the current and prior results differ. For example, if the currentand prior AI results differ by more than a threshold amount (e.g., bymore than a standard deviation or tolerance, etc.), then the changenotification 2750 is generated. The change notification can prompt apoint of care alert at the imaging device 1010, associated tablet orworkstation, RIS 2520 reading workstation 2640, etc.

Thus, certain examples enable an AI-driven comparison between currentand prior images and associated interpretations (e.g., change versus nochange, worse or better, progress or not, etc.). Additionally,information such as co-morbidities, patient demographics, and/or otherEMR data mining can be combined with image data to generate a riskprofile. For example, information such as patient demographics, priorimages, previous alert, co-morbidities, and/or current image can factorinto producing an alert/no alert, increase/decrease severity of alert,etc. In an example, a patient in the intensive care unit is connected toa ventilator, an oxygen meter, a blood pressure meter, an IV drip,and/or other monitor(s) while having images obtained. Additional datafrom connected meter(s)/sensor(s) can be combined with the image data toallow the AI model to better interpret the image data. A higherconfidence score and/or other greater degree of confidence can beassigned to an AI model prediction when more information is provided.Patient monitoring and/or other sensor information, patient vitals,etc., can be combined with prior imaging data to feed into an AIalgorithm model. Prior images and/or current images can be comparedand/or otherwise analyzed to predict a condition and/or otherwiseidentify a critical finding with respect to the patient.

Thus, certain examples help ensure and improve data and analysisquality. Providing an AI model on the imaging device 1010 enablesimmediate point-of-care action if the patient is critical or urgent. Insome examples, cloud-based access allows retrieval of other images forcomparison while still providing a local alert in real time at themachine 1010. Cloud access can also allow offloading of AI functionalitythat would otherwise be on the edge device 2610, machine 1010, broker2510, etc.

In certain examples, the broker 2510 can be used to intercept an imagebeing transmitted to the PACS 1044, and a prioritization message can beinserted to move an associated image exam up in the worklist.

In certain examples, a machine learning, deep learning, and/or otherartificial intelligence model can improve in quality based oninformation being provided to train the model, test the model, exercisethe model, and provide feedback to update the model. In certainexamples, AI results can be verified by reviewing whether anAI-identified anatomy in an image is the correct anatomy for theprotocol being conducted with respect to the patient. Positioning in theimage can also be evaluated to help ensure that organ(s)/anatomy are inview in the image that are expected for the protocol position, forexample. In certain examples, a user can configure his or her own usecase(s) for particular protocol(s) to be verified by the AI. Forexample, anatomy view and position, age, etc., can be checked andconfirmed before executing the AI algorithm model to help ensure qualityand clinical compliance.

In certain examples, a critical finding, such as a pneumothorax, isidentified by the AI model in the captured image data. For example, AIresults can indicate a likely pneumothorax (PTX) in the analyzed imagedata. In certain examples, feedback can be obtained to capture whetherthe user agrees with the AI alerts (e.g., select a thumbs up/down,specify manual determination, etc.). In certain examples, an audit trailis created to capture a sequence of events, actions, and/ornotifications to verify timing, approval, message routing/alerting, etc.

In certain examples, patient context can provide a constraint onapplication of an AI model to the available image and/or other patientdata. For example, patient age can be checked (e.g., in a DICOM header,via an HL7 message from a RIS or other health information system, etc.),and the algorithm may not be run if the patient is less than 18 yearsold (or a message to a user can be triggered to indicate that thealgorithm may not be as reliable for patients under 18).

With pneumothorax, for example, air is present in the pleural space andindicates thoracic disease in the patient. Chest x-ray images can beused to identify the potential pneumothorax near a rib boundary based ontexture, contour, pixel values, etc. The AI model can assign aconfidence score to that identification or inference based on thestrength of available information indicating the presence of thepneumothorax, for example. Feedback can be provided from users toimprove the AI pneumothorax detection model, for example.

FIG. 30 is a block diagram of an example processor platform 3000structured to executing the instructions of at least FIGS. 11-12 and28-29 to implement the example components disclosed and describedherein. The processor platform 3000 can be, for example, a server, apersonal computer, a mobile device (e.g., a cell phone, a smart phone, atablet such as an iPad™), a personal digital assistant (PDA), anInternet appliance, or any other type of computing device.

The processor platform 3000 of the illustrated example includes aprocessor 3012. The processor 3012 of the illustrated example ishardware. For example, the processor 3012 can be implemented byintegrated circuits, logic circuits, microprocessors or controllers fromany desired family or manufacturer.

The processor 3012 of the illustrated example includes a local memory3013 (e.g., a cache). The example processor 3012 of FIG. 30 executes theinstructions of at least FIGS. 11-12 and 28-29 to implement the systems,infrastructure, displays, and associated methods of FIGS. 1-27 such asthe image quality checker 1022, the pre-processor 1024, the learningnetwork 1026, the image enhancer 1028, the output 1030 of the processor1020/3012, the broker 2510, the edge device 2610, etc. The processor3012 of the illustrated example is in communication with a main memoryincluding a volatile memory 3014 and a non-volatile memory 3016 via abus 3018. The volatile memory 3014 may be implemented by SynchronousDynamic Random Access Memory (SDRAM), Dynamic Random Access Memory(DRAM), RAMBUS Dynamic Random Access Memory (RDRAM) and/or any othertype of random access memory device. The non-volatile memory 3016 may beimplemented by flash memory and/or any other desired type of memorydevice. Access to the main memory 3014, 3016 is controlled by a clockcontroller.

The processor platform 3000 of the illustrated example also includes aninterface circuit 3020. The interface circuit 3020 may be implemented byany type of interface standard, such as an Ethernet interface, auniversal serial bus (USB), and/or a PCI express interface.

In the illustrated example, one or more input devices 3022 are connectedto the interface circuit 3020. The input device(s) 3022 permit(s) a userto enter data and commands into the processor 3012. The input device(s)can be implemented by, for example, a sensor, a microphone, a camera(still or video, RGB or depth, etc.), a keyboard, a button, a mouse, atouchscreen, a track-pad, a trackball, isopoint and/or a voicerecognition system.

One or more output devices 3024 are also connected to the interfacecircuit 3020 of the illustrated example. The output devices 3024 can beimplemented, for example, by display devices (e.g., a light emittingdiode (LED), an organic light emitting diode (OLED), a liquid crystaldisplay, a cathode ray tube display (CRT), a touchscreen, a tactileoutput device, and/or speakers). The interface circuit 3020 of theillustrated example, thus, typically includes a graphics driver card, agraphics driver chip or a graphics driver processor.

The interface circuit 3020 of the illustrated example also includes acommunication device such as a transmitter, a receiver, a transceiver, amodem and/or network interface card to facilitate exchange of data withexternal machines (e.g., computing devices of any kind) via a network3026 (e.g., an Ethernet connection, a digital subscriber line (DSL), atelephone line, coaxial cable, a cellular telephone system, etc.).

The processor platform 3000 of the illustrated example also includes oneor more mass storage devices 3028 for storing software and/or data.Examples of such mass storage devices 3028 include floppy disk drives,hard drive disks, compact disk drives, Blu-ray disk drives, RAIDsystems, and digital versatile disk (DVD) drives.

The coded instructions 3032 of FIG. 30 may be stored in the mass storagedevice 3028, in the volatile memory 3014, in the non-volatile memory3016, and/or on a removable tangible computer readable storage mediumsuch as a CD or DVD.

From the foregoing, it will be appreciated that the above disclosedmethods, apparatus, and articles of manufacture have been disclosed tomonitor, process, and improve operation of imaging and/or otherhealthcare systems using a plurality of deep learning and/or othermachine learning techniques.

Thus, certain examples facilitate image acquisition and analysis at thepoint of care such as via a portable imaging device at the point ofpatient imaging. If images should be re-taken, further analysis doneright away, and/or other criticality explored sooner, rather than later,the example systems, apparatus, and methods disclosed and describedherein can facilitate such action to automate analysis, streamlineworkflow, and improve patient care.

Certain examples provide a specially-configured imaging apparatus thatcan acquire images and operate as a decision support tool at the pointof care for a critical care team. Certain examples provide an imagingapparatus that functions as a medical device to provide and/orfacilitate diagnosis at the point of care to detect radiologicalfindings, etc. The apparatus can trigger a critical alert for aradiologist and/or critical care team to bring immediate attention tothe patient. The apparatus enables patient triaging after the patient'sexam, such as in a screening environment, wherein negative tests allowthe patient to return home, while a positive test would require thepatient to be seen by a physician before returning home

In certain examples, a mobile device and/or cloud product enables avendor-neutral solution, proving point of care alerts on any digitalx-ray system (e.g., fully integrated, upgrade kit, etc.). In certainexamples, embedded AI algorithms executing on a mobile imaging system,such as a mobile x-ray machine, etc., provide point of care alertsduring and/or in real-time following image acquisition, etc.

By hosting AI on the imaging device, the mobile x-ray system can be usedin rural regions without hospital information technology networks, oreven on a mobile truck that brings imaging to patient communities, forexample. Additionally, if there is long latency to send an image to aserver or cloud, AI on the imaging device can instead be executed andgenerate output back to the imaging device for further action. Ratherthan having the x-ray technologist moved onto the next patient and thex-ray device no longer at the patient's bedside with the clinical careteam, image processing, analysis, and output can occur in real time (orsubstantially real time given some data transfer/retrieval, processing,and output latency) to provide a relevant notification to the clinicalcare team while they and the equipment are still with or near thepatient. For trauma cases, for example, treatment decisions need to bemade fast, and certain examples alleviate the delay found with otherclinical decision support tools.

Mobile X-ray systems travel throughout the hospital to the patientbedside (e.g., emergency room, operating room, intensive care unit, etc.Within a hospital, network communication may be unreliable in “dead”zones of the hospital (e.g., basement, rooms with electrical signalinterference or blockage, etc.). If the X-ray device relies on buildingWi-Fi, for example, to push the image to a server or cloud which ishosting the AI model and then wait to receive the AI output back to theX-ray device, then patient is at risk of not having reliability incritical alerts when needed. Further, if a network or power outageimpacts communications, the AI operating on the imaging device cancontinue to function as a self-contained, mobile processing unit.

Examples of alerts generated for general radiology can include criticalalerts (e.g., for mobile x-ray, etc.) such as pneumothorax, tubes andline placement, pleural effusion, lobar collapse, pneumoperitoneum,pneumonia, etc.; screening alerts (e.g., for fixed x-ray, etc.) such astuberculosis, lung nodules, etc.; quality alerts (e.g., for mobileand/or fixed x-ray, etc.) such as patient positioning, clipped anatomy,inadequate technique, image artifacts, etc.

Thus, certain examples improve accuracy of an artificial intelligencealgorithm. Certain examples factor in patient medical information aswell as image data to more accurately predict presence of a criticalfinding, an urgent finding, and/or other issue.

Certain examples evaluate a change in a clinical condition to determinewhether the condition is worsening, improving, or staying the sameovertime. For example, a critical result from a chest x-ray exam isconsidered to be a “new or significant progression of pneumothorax”, inwhich the radiologist shall call the ordering practitioner and discussthe findings. Providing an AI algorithm model on an imaging device withprior imaging examines enables the model to determine whether apneumothorax finding is new or significantly progressed and whether thefinding shall be considered critical or not.

Although certain example methods, apparatus and articles of manufacturehave been described herein, the scope of coverage of this patent is notlimited thereto. On the contrary, this patent covers all methods,apparatus and articles of manufacture fairly falling within the scope ofthe claims of this patent.

What is claimed is:
 1. An image processing apparatus comprising: amemory including first image data obtained in a first image acquisitionand instructions; and a processor to execute the instructions to atleast: evaluate the first image data with respect to an image qualitymeasure; when the first image data satisfies the image quality measure,process the first image data using a trained learning network togenerate a first analysis of the first image data; identify a clinicalfinding in the first image data based on the first analysis; compare thefirst analysis to a second analysis, the second analysis generated fromsecond image data obtained in a second image acquisition; and whencomparing identifies a change between the first analysis and the secondanalysis, trigger a notification at the imaging apparatus to notify ahealthcare practitioner regarding the clinical finding and prompt aresponsive action with respect to a patient associated with the firstimage data.
 2. The apparatus of claim 1, wherein first image acquisitionis associated with a protocol, and wherein the processor is to verifythat an anatomy associated with the protocol is in the first image data.3. The apparatus of claim 2, wherein the processor is to verify that aposition of the anatomy in the image is in compliance with the protocol.4. The apparatus of claim 1, wherein the trained learning network is toprocess the first image data using patient data from an electronicmedical record.
 5. The apparatus of claim 1, wherein the processor is toobtain the second analysis from a cloud-based system.
 6. The apparatusof claim 1, wherein the imaging apparatus includes an edge device toconnect to the cloud-based system.
 7. The apparatus of claim 1, furtherincluding a broker to connect the imaging apparatus to a healthinformation system.
 8. The apparatus of claim 7, wherein the healthinformation system includes a radiology information system, and whereinthe processor is to generate a prioritization message for the radiologyinformation system to prioritize an exam related to the patient as theresponsive action in response to the clinical finding.
 9. Acomputer-readable storage medium in an imaging apparatus includinginstructions which, when executed, cause at least one processor in theimaging apparatus to at least: evaluate the first image data withrespect to an image quality measure; when the first image data satisfiesthe image quality measure, process the first image data using a trainedlearning network to generate a first analysis of the first image data;identify a clinical finding in the first image data based on the firstanalysis; compare the first analysis to a second analysis, the secondanalysis generated from second image data obtained in a second imageacquisition; and when comparing identifies a change between the firstanalysis and the second analysis, trigger a notification at the imagingapparatus to notify a healthcare practitioner regarding the clinicalfinding and prompt a responsive action with respect to a patientassociated with the first image data.
 10. The computer-readable storagemedium of claim 9, wherein first image acquisition is associated with aprotocol, and wherein the instructions, when executed, cause the atleast one processor to verify that an anatomy associated with theprotocol is in the first image data.
 11. The computer-readable storagemedium of claim 10, wherein the instructions, when executed, cause theprocessor to verify that a position of the anatomy in the image is incompliance with the protocol.
 12. The computer-readable storage mediumof claim 9, wherein the trained learning network is to process the firstimage data using patient data from an electronic medical record.
 13. Thecomputer-readable storage medium of claim 9, wherein the instructions,when executed, cause the processor to obtain the second analysis from acloud-based system.
 14. The computer-readable storage medium of claim 9,wherein the imaging apparatus includes an edge device to connect to thecloud-based system.
 15. The computer-readable storage medium of claim 9,wherein the instructions, when executed, cause the at least oneprocessor to communicate with a broker to connect the imaging apparatusto a health information system.
 16. The computer-readable storage mediumof claim 15, wherein the health information system includes a radiologyinformation system, and wherein the instructions, when executed, causethe processor to generate a prioritization message for the radiologyinformation system to prioritize an exam related to the patient as theresponsive action in response to the clinical finding.
 17. Acomputer-implemented method comprising: evaluating, by executed aninstruction with at least one processor, the first image data withrespect to an image quality measure; when the first image data satisfiesthe image quality measure, processing, by executing an instruction withthe at least one processor, the first image data using a trainedlearning network to generate a first analysis of the first image data;identifying, by executing an instruction with at least one processor, aclinical finding in the first image data based on the first analysis;comparing, by executing an instructing with the at least one processor,the first analysis to a second analysis, the second analysis generatedfrom second image data obtained in a second image acquisition; and whencomparing identifies a change between the first analysis and the secondanalysis, triggering, by executing an instruction using the at least oneprocessor, a notification at the imaging apparatus to notify ahealthcare practitioner regarding the clinical finding and prompt aresponsive action with respect to a patient associated with the firstimage data.
 18. The method of claim 17, wherein first image acquisitionis associated with a protocol, and wherein the method further includesverifying that an anatomy associated with the protocol is in the firstimage data.
 19. The method of claim 18, further including verifying thata position of the anatomy in the image is in compliance with theprotocol.
 20. The method of claim 17, further including generating aprioritization message for a radiology information system to prioritizean exam related to the patient as the responsive action in response tothe clinical finding.